A Probabilistic Model for Knowledge Component Naming

Recent years have seen significant advances in automatic identification of the Q-matrix necessary for cognitive diagnostic assessment. As data-driven approaches are introduced to identify latent knowledge components (KC) based on observed student performance, it becomes crucial to describe and interpret these latent KCs. We address the problem of naming knowledge components using keyword automatically extracted from item text. Our approach identifies the most discriminative keywords based on a simple probabilistic model. We show this is effective on a dataset from the PSLC datashop, outperforming baselines and retrieving unknown skill labels in nearly 50% of cases. 1. OVERVIEW The Q-matrix, introduced by Tatsuoka [9], associates test items with attributes of students that the test intends to assess. A number of data-driven approaches were introduced to automatically identify the Q-matrix by mapping items to latent knowledge components (KCs), based on observed student performance [1, 6], using, e.g. matrix factorization [2, 8], clustering [5] or sparse factor analysis [4]. A crucial issue with automatic methods is that latent skills may be hard to describe and interpret. Manually-designed Q-matrices may also be insufficiently described. A data-generated description is useful in both cases. We propose to extract keywords relevant to each KC from the textual content corresponding to each item. We build a simple probabilistic model, with which we score keywords. This proves surprisingly effective on a small dataset obtained from the PSLC datashop. 2. MODEL We focus on extracting keywords from the textual content of each item (question, hints, feedback, Fig. 1). We denote by di the textual content (e.g. body text) of item i, and assume a Q-matrix mapping items to K skills ck, k = 1 . . .K. Figure 1: Example item body, feedback and hints. These may be latent skills obtained automatically or from a manually designed Q-matrix. For eack KC we build a unigram language model estimating the relative frequency of words in each KC [7]: